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The purpose of this paper is to provide a systematic procedure that uses 
evaluation data to improve an ongoing program during its in-progress stage. 

i 

RATIONALE 

The notion of educational evaluation evolved out of the context of the 
measurement discipline, and it was intimately related to the objectives of 
instruction and the improvement of instruction. 

Evaluation as a term came into being to express a broader concept than 
measurement. According to Pace (1968), evaluation puts measurement on the 
spot as being inadequate. He asserts that "evaluation accepted and welcomed 
the use of observations, interviews, check lists, questionnaires, testimony, 
the minutes of meetings, time logs, and many other means of assembling in- 
formation." Evaluation includes measurement and seeks to extend the range 
of measurement, but it is more than measurement. It includes psychometrics 
but holds that psychometric theory is irrelevant in many evaluation activities. 
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The use of educational evaluation is widespread. Current evaluation 

interest appears to be cutting across new "fronts". New technologies and new 

•r ' , 

problems give rise to corresponding evaluation. Now that education and 
educational involvements have expanded there is an accompanying need for the 
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application of the new evaluation concepts. According to Bloom (1968): 
"Evaluation is a two-edged sword which can enhance student learning and 
personality development." His position is that evaluation can have 
positive or negative effects on teachers, programs, school systems, 
administrators, and decision-makers in general. 



For the purpose of developing this methodology the following 

definitions of evaluation will be used: 

A procedure for collecting and analyzing data to produce pertinent 
information which can be used to facilitiate decision-making by 
decision-makers . 

Ultimately, the analysis of data aspect of the preceding definition 
offers information that influences the decision to maintain, or improve 
a program. It is the improvement aspects of the outcome of an evaluation 
that is the concern of this paper. 




PROCEDUR E 

Use the "Operationalization of Fuzzy Concepts"*, a methodology, 
to. generate the working set of variables. Consider a set of components 
or dimensions that represent the decision-maker’s goal intent. By using 
the suggested methodology, the assumption that the decision-maker’s real 
goal intent will be preserved throughout the process of generating the set 
components under consideration can be made. The words component, dimension, 
and variable cam be used interchangeably throughout the evaluation. 

The application of the operationalization of fuzzy concepts, a 
methodology, to a goal or a generalisation yields a list of behaviorally 
stated components that represent the decision-maker's real Intent of the 

*A syst'ematlc way to break down a generalization into Its component 
parts and provide a procedure for quantifying those components. 
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goal or generalization under consideration. The list of behaviorally 
stated components makes up the set of variables that dictate the eval- 
uation activity. 
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It should by stated, perhaps, that the operationalization of a 
fuzzy concept should not imply a point on a scale, rather, it should 
imply a variable (a component having numerical value assigned to it 
that is not fixed with respect to time) . Moreover, it is the variable that 
represents the source from which the evaluation activity comes. The concept 
of operationalization should be differentiated from things it purports 
to do. 

The evaluation activity Includes collecting, processing, and analyzing 
data. In particular, the evaluation activity Includes either a qualitative 
analysis of variables or a quantitative analysis of variables. The fact 
that the variables have been dellnes^ted to the extent that they can be 
stated in the form of a behavioral component implies the following: 

1. The desired behavior can be observed directly from a qualitative ^ 
perspective or; 

2. The desired behavior is measurable from a quantitative perspective. 
Thus giving rise to the development of either measurable techniques or 
observable techniques or both. 

INSTRUMENTATION 

In order to collect, process, or analyze data various tools and instru- 
ments should be used. Often times, tools such as statistics, data processing 
equipment, and computers are used. Such instruments as standardized tests, 
teacher-made tests, questionnaires, and "off-the-shelf” tests are often used 
also. . . , , 
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A variety of tools and instruments should not be used indiscriminantly . 
In fact, careful thought should be the underlying principle that leads to the 
decision of using a particular Instrument. The nature of the variables 
and the corresponding behaviors to be considered must influence the eval- 
uative Instrument to be used. 

The decision as to whether to use a ready-made Instrument should 
follow answers to a set of questions or concerns. They include the 
following; 

1. l^ether there is a ready-made instrument that provides answers 
to decision-makers concerns. 

a. If so, does it fit the desired use in its present form? 

b. If the acquisition of the instrument is economically 
practical. 

2. Uhat sacrifice does the use of a ready-made Instrument engender? 

3. What ill-effect will the potential environment have on the 
validity and reliability of a ready-made instrument? 

a. What aspects of the enterprise differ from those of the 
test criterion? 

b. What modification of the Instrument will be required by 
local policy? 

DATA 

Evaluation data should Influence systematic change during the in- 
progress stage of a program. Evaluation data must provide a rational 
basis for modification should program operations warrant such change. 

That is, if (a) a deviation from the real goal intent of the decison- 
maker occurs and (b) a discrepancy with respect to the implementation 
of the decision-maker's real goal Intent occurs, then, the evaluation 
data should provide implication for a systematic change that would lead 
to the attainment of the decision-maker's desired outcomes. The 
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principal factor that (a) indicates the existence of a deviation or 
discrepancy from the decision-makers real goal intent and (b) governs 
the implications for systematic change is the variable. 

A variable represents the smallest element of a goal intent. It 
is the variable, however, that serves a vital role in shaping the 
evaluation activity. Variables that have been operationalized — trans- 
formed into a non-fuzzy concept* or state — are the key factors that 
dictate the nature and form of the analytical aspect of the evaluation. 

The analysis of evaluation data may vary considerably with regards 
to instrumentation as mentioned in the preceding section. The instru- 
mentation is governed by the set of variables that characterize the 
decision-maker's real goal Intent. Thus, the analysis of data may include 
such, devices as anecdotal observations of specific states. For the 
most part, the evaluator has the responsibility of developing and applying 
procedures for describing and measuring the goal Intents truly and accurately 

Essentially there are two kinds of variables that the evaluator 
must accommodate. They are explicit and implicit. An explicit variable 
is an unambiguous characteristic of a goal Intent that can be observed 
directly. The following are examples of explicit variables: 



1. 


Pick up trash 




2. 


Be present li class 




3. 


Come to class on time 




4. 


Have a perfect attendance 




An 


implicit variable is an unambiguously paired 


characteristic of 


goal 


intent, usually combined with the conjunction 


'and' that can be 



*By definition a com’ept that cannot be observed directly is 
"fuzzy" concept. 
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observed directly. Some examples of implicit variables are the following: 

1. Run, skip, and jump 

2, Swing to and fro 

3„ Implement or execute 

4. Plain and simple 

This differentiation between variables is patently artificial. 

It is merely a convenient way of variable clarification, for no verb, 
or related set of verbs, for that matter, is in itself either explicit 
or implicit. In neither case is the distinction a hard and fast one. 

The explicit variable purports to represent a single behavior such 
as the examples cited. The implicit variable is a bit more complex 
as several related verbs represent a single behavior. Both explicit 
and implicit variables share a common feature, however. The specified 
behavior is a single activity which is common to the variables as 
indicated In the set of examples above. 

It is noteworthy to mention that, in general, this distinction 
between variables is a reflection of the inadequacy of this methodolgy 
to date. The limited range of evaluation methodology reinforces this 
distinction while highlighting its artificiality*. 



APPLICATION 

In order to apply evaluation data such that it can be used to 
Improve an ongoing program systematically, the evaluator and the 
decision-maker must work together closely. Moreover, their roles 
must be clearly distinctive. The role of the evaluator is to provide 
useful data for decision-making and, at the same time, preserve 
the real Intent of the decision-maker throughout the evaluation 



*The spirit of this paragraph and the preceding one was influenced 
from "Logic and Scientific Method in Research on Teaching" By M. Bradbeck. 
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of the program. The role of the decision-maker is, among other things, 
the management of the program throughout its developmental and operational 
stages . 



The operationalization of fuzzy concepts is a useful methodology for 
the evaluator to employ to generate a set of pertinent variables. Once a 
set of variables that represent the decision-maker's goal intent have been 
produced, the avaluator can biggin to influence programmatic changes for the 
sake of Imprcvement. In particular, he can do the following: 



1. Secure the dacision-naker ' s list of prioritized goal 
intentions . 

a. Have the decision-maker rank order the variables 
with respect to goals in terms of their importance 
to him. 

b. Consider the rank ordered set of variables for the 
purpose of developing observational techniques 
with regards to available resources. 

2. The decision-maker specifies the ordering of the variables 
to be evaluated. 

3. The evaluator evaluates each variable which manifest itself 
in the form of a sentence on its own merit. 

a. The predicate of the derived sentences dictates the 
evaluation activity. 

b. The required evaluation activity dictates the observation 
to be made or instrumentation to be used. 

4. The evaluator collaborates with the decision-maker about the 
implications and inferences as to the meaning of the variables 
under consideration. 

5„ The decision-maker considers the evaluators position with 

respect to the variables in question. In the case of deviations 
or discrepancies, the decision-maker should either: 

a. Modify the program, or 

b. Modify the goals and, hence, the variables. 

6. The evaluator develops appropriate observatipnal techniques. 

a. In the case of direct and indirect ob^trusive or unobtrusive 
measures, the evaluator develops the technique. 
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b. In the case of instrumentation the evaluator prescribes, 
describes, or secures, and/or develops the appropriate 
instrumient . 

c. In the case of appropriate ready-made instruments, the 
evaluator endorses them. 

7. The evaluator has tie responsibility of carrying out the 
observational techniques and/or executing ;he instrumentation. 

a. In the case of observational techniques, verbs such as 
pick, come, skip, and Jump dictate the evaluation activity. 

b. In the case of developing instruments, a clear statement 
that the instrument is intended for thii evaluation of a 
particular program only should be prominently stated in 
the accompanying instruments. The degree to which such 
a statement can be e^rpectea to restrict the use of such 
Instruments will vary. 

8. The evaluator reports his findings to the decision-maker — 
he states his conclusions in wiriting. 

The preceding steps characterize much of the way in which an 
evaluator evaluates an ongoing program. It is interesting to observe, 
however, that only one of the above steps is crucial to improving a 
program during its in-progress stage so that desired results will be 
a natural outcomer In particular, step 5 is critical if evaluation 
data Is to be used methodically to Improve an ongoing program. 
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The implications of step 5 provide the decision-maker with the 
choice of either modifying the program or modifying the goals. If the 
decision-maker decides to modify the program, the evaluated can be 
a useful resource to aid in improving the ultimate outcome of the 
program's operations. If the decision-maker decides to change the 
goals so that the variables are consistent with the program's oper- 
ations then the job of the evaluator changes. That is, he no longer 
has the responsibility of evaluating the program that was under con- 
sideration originally. A change of goals implies a change of intended 
outcomes. 
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SUMMARY 



The re-emergence of evaluation of the 1960 's continues its upward 
trend across many educational programs and social programs. Thus, the^ 
way in which evaluation data can and must be used to effect programmatic 
change needs to be made available to perspective users. This writer 
has offered a rationale and a procedure for using evaluation to improve 
an ongoing program. 

In order to use evaluation data to improve an ongoing program, the 
evaluator must secure a commitment fron the decision-maker to work with 
him throughout much of the evaluative process. That is, the evaluator 
must secure and maintain the decision-maker's cooperation, support, 
and participation throughout the development of observational techniques 
and/or Instruments., The evaluation of an enterprise cannot be conducted 
adequately in the ^lbsence of the decision-maker. In fact, good evaluations 
are not merely carried out for decision-maker's. They are in part carried 
out by decision-makers. 

The matter of having decision-makers play a vital role in the 
evaluation of an enterprise suck as an ongoing program should be, in part, 
b.y design. It is the belief of this writer that the decision-maker's 
involvement will accomplish the following: 

1. Increase the decJslon-maker *s awareness of the evaluation 
endeavor . 

2. Develop within the declaion-maker respect and appreciation 
for the evaluative results. 

3. Proylci? the decision-maker the wherewithal to use evaluative 
data to Improve the program. 

4. Increase the likelihood of the decision-maker using the 
evaluation results. 

In addition to having designated a role for the decision-maker. 
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this writer points out how the evaluator and the decision-maker should 
work together and, at the same time, maintain distinct roles. It is 
incumbent upon the evaluator to keep the decision-maker informed of any 
deviations and/or discrepancies. Consequently, the importance of step 5 
of the application section cannot be over emphasized. It is step 5 that 
makes it possible for the decision-maker to take advantage of the eval- 
uation data during the in-progress stage of the program and make 
modifications effectively, these modifications should lead to programmatic 
improvements that are consistent with the goal intent of the decision-maker. 
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